Multiple-choice items are easy to administer from a technical standpoint, but are often difficult and time-consuming to write because they require a comprehensive understanding of all possible responses that a participant might give. For questions with a single correct answer, this means anticipating all likely wrong answers based on a deep understanding of all the misconceptions and misunderstandings that might lead a participant to respond incorrectly. Even for questions without a single correct answer (e.g., attitudinal questions), it can be difficult to predict all possible opinions that a participant might have when faced with the item prompt. Developing response options for multiple-choice items is challenging enough to merit numerous sets of published guidelines and rules; see, e.g., Haladyna, Downing, & Rodriguez, “A review of multiple-choice item-writing guidelines for classroom assessment,” Applied Measurement in Education, 15(3), 309-333 (2002).
In contrast, asking participants to respond to free-response items instead of multiple-choice items solves this particular problem but presents its own set of difficulties. Contrasted with supervised learning analysis, which may involve either an algorithmic definition or a training set of classified entries, the underlying clustering of the unsupervised learning for the participant responses is unknown. In addition, some rare types of free-response items can be scored automatically (see, e.g., Attali & Burstein, “Automated essay scoring with e-rater v.2,” Journal of Technology, Learning, and Assessment, 4(3), 1-31 (2006); Bennett, Morley& Quardt, “Three Response Types for Broadening the Conception of Mathematical Problem Solving in Computerized Tests,” Applied Psychological Measurement, 24(4), 294-309 (2000)), however, the time and monetary costs to set up and use such a system can make them infeasible. Furthermore, without a way of automatically scoring or classifying item responses, an instructor cannot process dozens or hundreds of participant responses in real time to dynamically adjust instruction, as is the goal with any formative assessment technique.
Consequently, there is a need for an approach using unsupervised learning that automatically classifies or scores participant responses to open-ended items and provides real-time feedback to the instructor.